Papers with Masked Language Models
A Benchmark Evaluation of Clinical Named Entity Recognition in French (2024.lrec-main)
Copied to clipboard
| Challenge: | Masked Language Models (MLMs) have shown strong performance on many NLP tasks. |
| Approach: | They evaluate masked language models for biomedical French on the task of clinical named entity recognition using gold-standard corpora. |
| Outcome: | The proposed model outperforms standard models on the task of clinical named entity recognition in biomedical French while remaining lighter than current models. |
Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging? (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent Part-Of-Speech (POS) induction models assume certain independence assumptions that do not hold in real languages. |
| Approach: | They propose a Masked Part-of-Speech Model (MPoSM) that can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction. |
| Outcome: | The proposed model can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction. |
Gender Bias in Masked Language Models for Multiple Languages (2022.naacl-main)
Copied to clipboard
| Challenge: | Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages. |
| Approach: | They propose to use English attribute word lists to evaluate bias in eight languages without manually annotating data. |
| Outcome: | The proposed model significantly correlates with the existing English datasets for gender bias. |
Pedagogically Aligned Objectives Create Reliable Automatic Cloze Tests (2024.naacl-long)
Copied to clipboard
| Challenge: | a recent study has focused on the cloze training objective of Masked Language Models . distractors must be distinct and incorrect, and can be biased if the test creator is testing two versions of a text . |
| Approach: | They propose a method that jointly optimizes sets of distractors from Masked Language Models. |
| Outcome: | The proposed method has stronger correlation with teacher-created comprehension tests than state-of-the-art neural method and is more internally consistent. |
QueerGen: How LLMs Reflect Societal Norms on Gender and Sexuality in Sentence Completion Task (2026.findings-eacl)
Copied to clipboard
| Challenge: | Autoregressive Language Models (ARLMs) partially mitigate these patterns, while closed-access ARLMs tend to produce more harmful outputs for unmarked subjects. |
| Approach: | They examine whether explicit information about a subject’s gender or sexuality influences LLM responses across three subject categories: queer-marked, non-queer-mark, and the normalized "unmarked" category. |
| Outcome: | The proposed models reproduce normative social assumptions, but the form and degree of bias depend on model characteristics, which may redistribute—but not eliminate—representational harms. |
MLMLM: Link Prediction with Mean Likelihood Masked Language Model (2021.findings-acl)
Copied to clipboard
| Challenge: | Knowledge Bases (KBs) are easy to query, verifiable, and interpretable. however, they scale with man-hours and high-quality data. |
| Approach: | They propose to commit the knowledge embedded in MLMs to a KB, making it interpretable . they propose to use a mean likelihood Masked Language Model to compare the likelihood of generating different entities to perform link prediction in a tractable manner. |
| Outcome: | The proposed approach compares the likelihood of generating different entities to perform link prediction in a tractable manner. |
Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit remarkable capabilities in zero-shot and few-shot settings, but they struggle with extending to few- shot and zero- shot settings due to their architectural design. |
| Approach: | They propose a technique that models discriminative tasks as a set of finite statements and trains an encoder model to discriminate between the potential statements to determine the label. |
| Outcome: | The proposed method achieves competitive performance compared to state-of-the-art LLMs with significantly fewer parameters. |
Masked Audio Text Encoders are Effective Multi-Modal Rescorers (2023.findings-acl)
Copied to clipboard
| Challenge: | Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition systems. |
| Approach: | They propose a multi-modal masked language model rescorer which integrates acoustic representations into the input space of MLM. |
| Outcome: | The proposed model reduces word error rate (WER) by 4%-16% on in-domain and 3%-7% on out-of-domain datasets over the text-only baseline. |
NB-MLM: Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-training Masked Language Models (MLMs) on massive datasets is expensive, but it is performed for each domain or task individually and is resource-demanding. |
| Approach: | They propose a method for more efficient adaptation that focuses on predicting words with large weights of the Naive Bayes classifier trained for the task at hand. |
| Outcome: | The proposed method improves sentiment analysis by focusing on predicting words with large weights of the Naive Bayes classifier trained for the task at hand. |
[MASK]ED - Language Modeling for Explainable Classification and Disentangling of Socially Unacceptable Discourse. (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained PMLMs often degrade in multi-label SUD classification due to overlapping linguistic cues. |
| Approach: | They propose an artifact-guided pre-training strategy that injects statistically salient linguistic features into masked language models to enhance generalization across discourse types. |
| Outcome: | The proposed approach improves performance in 10 datasets extensively used in SUD classification benchmarks. |
CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language model scores are often treated as probabilities, but their reliability as probability estimators has mainly been studied through calibration, overlooking other aspects. |
| Approach: | They propose a framework to assess model reliability across interchangeable completion and conditioning orders by performing statistical tests on real and synthetic data to eliminate training effects. |
| Outcome: | The proposed framework assesses the consistency of model predictions across interchangeable completion and conditioning orders on real and synthetic data to eliminate training effects. |
Evaluating Short-Term Temporal Fluctuations of Social Biases in Social Media Data and Masked Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Social biases such as gender or racial biase are reported in language models . a recent study has shown that MLMs encode discriminatory social biase . |
| Approach: | They analyse temporal corpora of MLMs trained on chronologically ordered temporal snapshots . they find that gender and racial biases are encoded in MLM models . |
| Outcome: | The proposed model identifies gender biases in MLMs but most remain stable over time . gender bias is associated with higher likelihood scores in some demographic groups . |